Skip to content

fix(flaky): propagate auth trigger errors#217

Merged
zeshi-du merged 1 commit into
TestSprite:mainfrom
JerryNee:fix/flaky-auth-propagation
Jul 18, 2026
Merged

fix(flaky): propagate auth trigger errors#217
zeshi-du merged 1 commit into
TestSprite:mainfrom
JerryNee:fix/flaky-auth-propagation

Conversation

@JerryNee

@JerryNee JerryNee commented Jul 9, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

Replaces accidentally closed PR #200.

Updates test flaky so auth failures while triggering reruns are treated as fatal API errors instead of being recorded as ordinary flaky attempts.

Specifically, this propagates these trigger-time auth errors:

  • AUTH_REQUIRED
  • AUTH_INVALID
  • AUTH_FORBIDDEN

That keeps the CLI from reporting an auth/environment failure as a flaky test signal.

Related issue

Fixes #199

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Testing

Previously validated on the original PR branch with:

  • npm test -- src/commands/test.flaky.spec.ts
  • npm run typecheck
  • npm run format:check
  • npm run lint
  • NO_COLOR= npm test

Notes for reviewers

Focused change in runFlaky: auth-related rerun trigger errors are rethrown, while normal per-attempt flaky recording remains unchanged.

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of authentication failures when rerunning flaky tests.
    • Error responses now surface correctly for missing, invalid, or forbidden auth, with a clear exit code and no extra output.
    • Failed rerun attempts are no longer counted as successful trigger retries in these auth error cases.

@coderabbitai

coderabbitai Bot commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 61aecc34-3f8d-4cc2-8d77-5fd5582036cc

📥 Commits

Reviewing files that changed from the base of the PR and between 3305dfa and 6d9eb7c.

📒 Files selected for processing (2)
  • src/commands/test.flaky.spec.ts
  • src/commands/test.ts

Walkthrough

This PR updates the test flaky command so that authentication-related errors (AUTH_REQUIRED, AUTH_INVALID, AUTH_FORBIDDEN) raised during rerun trigger calls are re-thrown as fatal errors instead of being recorded as non-fatal flaky attempts. Corresponding mock support and tests are added.

Changes

Auth failure propagation in test flaky

Layer / File(s) Summary
Fatal auth error classification and re-throw
src/commands/test.ts		 Adds isFlakyFatalTriggerError helper to detect auth-related ApiError codes and re-throws them in runFlaky's triggerRerun catch block instead of recording a non-fatal attempt.
Mock auth responses and test coverage
src/commands/test.flaky.spec.ts		 Adds TriggerAuthErrorCode type, extends makeFlakyFetch mock options with triggerAuthError to simulate 401/403 auth failures on rerun trigger, and adds a parameterized test verifying ApiError propagation, exitCode: 3, no stdout, and no rerun triggers.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant runFlaky
  participant triggerRerun
  participant isFlakyFatalTriggerError
  runFlaky->>triggerRerun: POST runs/rerun
  triggerRerun-->>runFlaky: throws ApiError(code)
  runFlaky->>isFlakyFatalTriggerError: check code
  alt AUTH_REQUIRED/AUTH_INVALID/AUTH_FORBIDDEN
    isFlakyFatalTriggerError-->>runFlaky: fatal=true
    runFlaky-->>runFlaky: re-throw ApiError, exitCode=3
  else other error
    isFlakyFatalTriggerError-->>runFlaky: fatal=false
    runFlaky-->>runFlaky: record non-fatal attempt
  end
Loading

Possibly related PRs

Suggested reviewers: ruili-testsprite, zeshi-du

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: propagating auth trigger errors in flaky test handling.
Linked Issues check ✅ Passed The PR rethrows auth-related rerun trigger errors and adds coverage, matching the linked issue's required fatal auth behavior.
Out of Scope Changes check ✅ Passed The changes stay focused on flaky rerun auth-error handling and related tests, with no obvious unrelated additions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

zeshi-du
zeshi-du approved these changes Jul 18, 2026
View reviewed changes

@zeshi-du zeshi-du left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed the bug is live on main (every trigger error including auth is scored as a flaky attempt), and the fix rethrows AUTH_* before scoring — exit 3, empty stdout, zero recorded attempts, with the flaky-cap and cli-source trigger call untouched. Thanks @JerryNee!

@zeshi-du
zeshi-du merged commit eaf0585 into TestSprite:main Jul 18, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zeshi-du zeshi-du zeshi-du approved these changes
@ruili-testsprite ruili-testsprite Awaiting requested review from ruili-testsprite ruili-testsprite is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Hackathon] test flaky should propagate auth failures instead of scoring them as flaky

2 participants

@JerryNee @zeshi-du